The Case for Multisystemic Therapy: Evidence or Orthodoxy?

In this paper, I respond to comments by Henggeler, Schoenwald, Borduin, and Swenson (this issue) on my recent article, “Lessons from a systematic review of Multisystemic Therapy.” I identify factual and logical errors in their response, show how relevant research has been misinterpreted and misrepresented, and suggest constructive new directions for Multisystemic Therapy and the evidence-based practice movement. Evidence or orthodoxy? 3 Introduction Sometimes science matters. New evidence or a new look at old evidence can disconfirm even hallowed assumptions. It is thus entirely reasonable to question whether evidence for the effectiveness of Multisystemic Therapy (MST) holds up under close scrutiny. Such an appraisal is warranted especially when rigorous, empirically-based techniques of research synthesis might usefully revise findings of earlier narrative reviews. In “Lessons from a systematic review of Multisystemic Therapy” (Children and Youth Services Review, 27, 445-463), I indicated that the effectiveness of MST remains in doubt. The effectiveness of MST was not the main subject of that article, however. In the “Lessons” article, I identified wide gaps between the science and practice of research synthesis and argued for enhanced rigor in the conduct, reporting, and synthesis of controlled trials. To illustrate my case, I used examples from my research team’s systematic review and meta-analysis of controlled studies on the effects of Multisystemic Therapy, but as its title suggests the “Lessons” article was not intended to be a full account of that systematic review (for a full report, see Littell, Popa, & Forsythe, 2005). Scott Henggeler, Sonja Schoenwald, Charles Borduin, and Cynthia Swenson chose to respond to selected portions of “Lessons” in “The Littell paper: Methodological critique and meta-analysis as Trojan Horse” (this issue). My purpose in the present paper is to identify and correct the factual and logical errors in their response, and to suggest constructive directions for MST and the evidence-based practice movement. But it is important to consider first the context of this debate. Dr. Henggeler and his coauthors are the developers of MST. They also served as principal investigators on the major federal grants used to test this intervention. By 2003, research grants for MST totaled $35 million (Henggeler, 2003). In January 2004, Dr. Henggeler Evidence or orthodoxy? 4 (2004a) announced the receipt of approximately $20 million in new research grants. Dr. Henggeler and his associates direct and hold stock in MST Services, Inc., a private consulting firm “which has the exclusive licensing agreement... for the dissemination of MST technology and intellectual property” (Rowland & Halliday-Boykins, 2004, p. 4). MST Services Inc. collects licensing, training, and consulting fees of approximately $400 to $550 per youth served (Strengthening Families America, 1999) and serves about 10,000 families per year (Henggeler, 2003). Quite apart from their hard-won professional pride in their achievements, the developers of MST have a financial interest in this “model program.” There is nothing intrinsically wrong with this, but the potential for conflict of interest is obvious and bears watching. As the developing and marketing of pharmaceuticals has taught us, independent trials and disinterested, rigorous, systematic reviews of all credible evidence should be essential elements of policy making for the public good. I have no personal or financial stake in MST or any other intervention model. I have never met Dr. Henggeler or any of his colleagues. Contrary to their suggestions, if there is a “camp” with some “insidious strategy” to “camouflage the commitment to the status quo” (Henggeler et al., this issue), I have nothing to do with it. I take pride in being a member of the “disputatious community of ‘truth-seekers’” that Donald Campbell (1988) envisioned. I have no other agenda, no secret weapons – no Trojan Horse. Indeed, I share the concerns of Henggeler and his colleagues, and scholars such as Leonard Bickman (2002), about the lack of evidence for the effectiveness of most services for children and families – what Henggeler and colleagues called the “cottage industry” of mental health services. However, it is the “MST industry” that is the subject of this debate. Will the Evidence or orthodoxy? 5 MST industry utilize independent evaluations and evidence-based critiques to advance knowledge and practice on behalf of youth and families, or will it advance a new orthodoxy and demonstrate the “distrust, fear, and behavior protective of the status quo” of which Henggeler and his colleagues are so critical? I return to these broader issues by way of conclusion. Immediately below, I consider matters of fact and logic. Factual errors in Henggeler et al.’s response Contrary to Henggeler et al.’s assertions, I drew no “conclusions” about the effects of MST in the “Lessons” article. Instead, I characterized preliminary findings as such and noted that these results might change as new information became available. The suggestion that my papers were “broadly distributed through the Internet and media” is false. As a professional courtesy, I sent preliminary results of our systematic review to Henggeler and his colleagues, independent MST investigators, and reviewers whose work we cited, noting that this was a draft for comment and discussion only. Like Henggeler, his colleagues, and other scholars, I actively seek input on my work at scholarly and professional conferences and I make conference presentations available on request. The claim that statements in my paper were not “substantiated with data” is false. I provided Henggeler and colleagues with a full report on our meta-analysis -and most of the data in it were generated by them. The claim that our systematic review was not peer-reviewed is false. After preliminary results were vetted by stakeholders and experts, and beginning in early January 2005, the final report was assessed by ten anonymous readers for the Cochrane Collaboration and the Campbell Collaboration. Evidence or orthodoxy? 6 The claim that I “ignored” feedback is false. I provided Henggeler and colleagues with detailed written responses to their criticisms on October 20, 2003, December 2, 2003, and February 11, 2005. I disagree with their views of the role of treatment fidelity and site-level variations in explaining results. These issues are addressed in our Cochrane review (Littell, Popa, & Forsythe, 2005) and are taken up below. To disagree with others is not to ignore them. I did not suggest that the MST trials “were not credible” because of variable follow-up periods (or for any other reason), nor did I imply that the length of follow-ups was “arbitrary” or “manipulated” (Henggeler et al., this issue). If the studies were not credible, why would we include them in our review? I did not ignore “the efficacy and effectiveness results generated in numerous...MST trials” (Henggeler et al., this issue). These data are included in our systematic review. My research team did not use a study quality rating system to weight results of our metaanalysis. We did not privilege or elevate results of any study. Logical errors Equating the purposes of primary and secondary studies Henggeler et al. claim that I “did not consider distinctions among purposes” of various MST trials (this issue). However, they did not recognize that, like any secondary analysis of data, a systematic review and/or meta-analysis may be conducted for purposes other than those of the primary studies under review. For example, one can examine gender differences using data from studies that were not originally intended to address this issue. The purposes of my team’s systematic review were twofold: We aimed to provide unbiased estimates of the overall (i.e., 1 Here the term “systematic review” means a review of available data that uses explicit inclusion and exclusion criteria, a systematic strategy to locate all potentially-relevant studies, inter-rater agreement on study inclusion decisions, and systematic coding and inter-rater agreement on key features of included studies. Meta-analysis refers to quantitative synthesis of results of multiple studies. Meta-analysis may or may not be part of a systematic review. Evidence or orthodoxy? 7 average) effects of MST in controlled studies of outcomes for youth and families and, to the extent possible, identify sources of heterogeneity (i.e., moderators) of these effects across studies. For reasons discussed below (and in our systematic review), available data do not support systematic moderator analysis; thus, it is not possible at present to determine why some studies reported greater effects than others. All of the studies in our review assessed effects of MST, and that is sufficient for our primary purpose, which mirrors proponents’ claims that MST is effective across several service populations and settings (Henggeler, Schoenwald, Rowland, & Cunningham, 2002). Post hoc explanations for heterogeneous effects Any explanation for variations in results across MST studies is purely speculative at this point. The MST trials differ in terms of their sample characteristics, comparison conditions, perceived fidelity to MST, investigator independence, and the extent to which they support intent-to-treat analysis. Any of these factors can be used to explain the fact that results of early studies conducted by Henggeler and his associates were not replicated in the large, independent, multi-site Ontario trial (Leschied & Cunningham, 2002); yet we cannot know which factors were responsible for this difference because plausible sources of heterogeneity are confounded. Nevertheless, Henggeler and colleagues advanced several post hoc explanations for these differences, which I consider next. Efficacy, effectiveness, and transportability Henggeler (2004b) suggested that results of early MST efficacy trials were more promising than those of later studies of effectiveness, and MST developers took me to task for not recognizing this “difference.” However, the distinctions between efficacy and effectiveness research are not at all clear in the original MST studies. Referring to the original MST trials, Evidence or orthodoxy? 8 Schoenwald and colleagues said the designs “could be considered hybrids of ‘efficacy’ and ‘effectiveness’ research” (Schoenwald, Sheidow, Letourneau, & Liao, 2003, p. 234). Post-hoc distinctions on these matters are debatable. Henggeler and colleagues refer to the Ontario trial as a transportability study, but according to its investigators, this study was designed to assess effectiveness and efficiency and it did not include plans for a systematic study of factors that affect transportability (see Leschied & Cunningham, 2002).